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@ There can be a wrong decision about the motion 
direction for a large number of pixels when motion 
vectors are found by the block matching method. A 
picture for which motion vectors are already deter- 
mined by e.g. block matching, is according to the 
invention first preprocessed. The object is to in- 
crease the contrast in the picture and to make the 
form and contours of the surfaces clearly discern- 
ible. Then the picture is quantified in order to de- 
crease the number of gray levels, and then the 
picture Is filtered. The result is a picture comprising 
uniformly shaded segments of different sizes and 
forms. One block can contain several segments. 
Each segment gets a suitable motion vector of its 
own, taking into account the known motion vectors of 
the blocks. 
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The invention relates to a nnethod which is able 
to determine the nnotion vectors in smaller seg- 
ments than blocks of a television picture, when the 
motion vectors of the blocks are known. 

Information of the motion contents of the pic- 
ture is added to the transmitted signal so that in a 
television receiver the received video signal can be 
processed by some motion compensated method, 
by increasing the line rate of the television picture 
(LRU, Line Rate Upconversion) or by increasing the 
field rate (Field Rate Upconversion). In high defini- 
tion television (HDTV) systems the picture is di- 
vided into small blocks, e.g. 16*16 pixels, and by a 
suitable motion detection method we determine for 
the blocks a motion vector which defines the direc- 
tion and extent of the motion. The motion vectors 
are transmitted as data signal separated from the 
video signal. At the transmission end the blocks 
having different motion contents are processed dif- 
ferently, whereby each transmitted block is accom- 
panied by a certain motion vector. When the re- 
ceiver generates the original picture it can in then 
select the correct signal processing method in ac- 
cordance with the motion contents of the block. 

There are several known motion detection 
methods to calculate the motion vectors. Most of- 
ten these methods are based on difference signals, 
which are calculated between two pictures, and on 
whether the picture contains motion or not, which is 
concluded with the aid of these difference signals. 
The motion estimation methods may be divided 
into two main classes: methods which detect mo- 
tion parameters, and methods which measure pixel 
speed. Further it is possible to classify the pixel 
measuring methods in methods based on spatial- 
temporal differentials, methods based on matching 
and examining block equivalence at different mo- 
ments, and methods based on Fourier techniques. 

The motion parameter difference methods are 
not particularly well suited to estimate the motion in 
television, due the limitation placed on the picture 
and due to the method itself. Techniques based on 
spatial-temporal differentials which belong to the 
methods measuring the pixel speed are based 
solely on propagation models. They assume that 
the intensity variation is entirely a linear function of 
the movement in the television picture field. Sev- 
eral differentials are calculated both in the temporal 
direction (between successive fields) and in the 
spatial direction (in the horizontal and vertical direc- 
tions). These differential ratios provide the spatial 
movement, the pixel being the unit for each picture. 
The Fourier techniques use correlation of two pic- 
tures, so that a two-dimensional Fourier transforma- 
tion is first made for each picture, then the cor- 
responding frequency components are multiplied, 
and then the inverse transformation is performed. 
The result is a correlation surface having a peak at 



those coordinates which correspond to a move- 
ment between two pictures. The Fourier method is 
well suited for measuring the global motion, and 
the accuracy can be further increased by using 

5 some further processing, such as interpolation of 
the correlation surface and postprocessing. 

An often used transformation belonging the 
methods measuring the pixel speed is block 
matching, which estimates area by area instead of 

10 estimating pixel by pixel. Below the word "field" 
means both the field of an interleaved picture and 
the field of a progressive picture, which then is 
equal to the picture. In block matching the fields 
are divided into blocks of the size n*m, and it is 

75 assumed that all pixels within the block have equal 
movement vectors. An example illustrates the block 
matching: let's assume that a block (= a sub-area 
of a certain size In the picture) of n"m pixels In a 
field t contains an object, e.g. a picture of a car. 

20 This block may be called the reference block. The 
car moves, and in the next field t + 1 it has moved 
into a new position. 

Now we must find the new position. This is 
made so that in the new field t + 1 we define a 

25 search area within which we move a block having 
the same size as the reference block, i.e. n*m 
pixels. We can assume this block to be a window, 
through which we see a pixel set of the field within 
an area having the size of the window. In each 

30 position of the contents of the block (window) is 
compared with the contents of the reference block 
of the field t. In the comparison we compare the 
pixels of the window with the corresponding pixels 
of the reference block of field t. As the block 

35 (window) is moved over the field t + 1, then the 
comparison condition is met in some position of 
the field t + 1, which then is interpreted so that the 
blocks have the same contents. Now the blocks 
"match", because their contents are identical. Then 

40 we know that the car has moved to this position in 
the field t + 1 . Thus we obtain the magnitude and 
direction of the movement. 

Block matching is a motion vector determina- 
tion method which is used very often due to its 

45 clarity. In this method, as well as in the other above 
mentioned methods, the motion vector determined 
for a certain block is assumed to represent the 
motion direction of all pixels in the block, and the 
picture processing is made according to this as- 

50 sumption. Thus the motion vectors only estimate 
the motion of the majority of the pixels in the block, 
or in some cases they only minimize a certain error 
criterion. Thus the real movement within a block 
seldom follows the boundaries of the block but 

55 crosses them in an arbitrary way. In certain cases 
this way to set the direction of the motion vector of 
a block to be the same over the whole block will 
thus cause clearly discernible picture errors. 
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Figure 1 shows in an illustrative way the reason 
why picture errors are created when the prior art 
nnotlon vector determination is used. The figure 
shows four regular quadratic blocks A, B, C and D, 
each block having a respective motion vector de- 
termined by the block matching method. 

The motion vectors are shown by arrows in 
each block, and according to these the movement 
direction of all pixels in blocks A, B and C is to the 
left, whereas the movement direction of all pixels in 
block D is up and to the right. According to prior 
art all pixels of a block get the same direction of 
the motion vector. As we can see In the figure, the 
direction of the step line is probably up and to the 
right, because that is the direction of the majority 
of the pixels in block D. Thus the directions of the 
segments divided by the step line do not follow the 
borders of the blocks (they are not the same as the 
direction of the motion vector of the block), but 
crosses the borders. The light segments in blocks 
A, B and C have a correct direction of motion. Also 
the darkened segment in block D has the correct 
direction of motion. But on the contrary, the correct 
direction of motion of the striped segments in 
blocks C and D is up and to the right, and not to 
the left. Thus we can see that the prior art decision 
concerning the motion direction is erroneous for a 
large part of the pixels, or for those pixels belong- 
ing to the striped areas. Therefore discernible er- 
rors are caused when the picture is processed 
block by block in the receiver according to these 
main vectors. 

The object of this Invention Is to obviate the 
disadvantages of the known methods to determine 
the motion vectors, and thus to obtain a method 
with which it is possible to determine the motion 
vectors correctly also for small segments of Irregu- 
lar shape which border on the edges of moving 
objects. 

This object is achieved with the means de- 
scribed in the characterizing clause of claim 1 . 

The method is shown as a block diagram in 
figure 7. 

First we preprocess the picture, for which mo- 
tion vectors already have been determined by a 
known method, e.g. by block matching. The object 
of this is to increase the contrast in the picture and 
to make contours clearly discernible. The object Is 
thus to detach the objects from their background 
on the basis of the spatial Information only, and to 
make different areas of the picture clearly discern- 
ible. The areas are distinguished by their gray 
level. 

Then the number of gray levels is reduced by 
thresholding the picture samples, e.g. from sam- 
ples of 8 bits to samples of 3 bits, whereby the 
number of gray levels is reduced from 256 to 8. 
The thresholding can be equidistant and the num- 



ber of gray levels even smaller, e.g. four levels (2 
bits) Is quite suitable. After thresholding the picture 
seems to comprise areas of different size and form 
but of uniform shade, whereby these areas are 

5 distinguished by their gray level. These areas are 
called segments. The higher the number of levels 
after thresholding the more there are small discern- 
ible areas in the picture representing the details of 
the picture. The object of the thresholding is on 

10 one hand to get rid of very small segments, and 
thus a low level thresholding with e.g. 4 levels is 
preferable. After the thresholding there is still a 
postprocessing, with which the picture Is smoothed 
even more by eliminating islands of the size of a 

75 few pixels which strongly differ from their neighbor- 
hood. As a result of the above mentioned mea- 
sures the picture is simplified to areas or segments 
of different sizes but with a uniform shade (uniform 
color), and the number of the gray levels (colors) in 

20 the picture is minimal. A segment is thus a uniform 
set of adjacent pixels having the same gray scale 
value after the preprocessing, quantizing and post- 
processing. 

Then a motion vector is determined for each 

25 segment. As a basis there is the fact that the same 
picture is divided into blocks (adjacent picture 
areas of equal size) in the transmitter or the en- 
coder, a block motion vector being determined for 
each block by a known method, such as by block 

30 matching. In a way each block covers a part of the 
segmented picture. If a block border crosses a 
segment, then the segment Is thought of as being 
divided in two, whereby one part belongs to one 
block and the other part to another block. Thus the 

35 segments can not extend into the area of a neigh- 
boring block. The largest segment in the area of a 
block Is found first, and It Is called the main seg- 
ment. Thus the area of a block can contain seg- 
ments of four different kinds: 

40 1) one main segment; 

2) one or more segments being not the main 
segment and not next to a segment of a neigh- 
boring block; 

3) one or more segments being not the main 
45 segment and being next to the main segment of 

a neighboring block; 

4) one or more segments being not the main 

segment and being next to a segment of a 
neighboring block, which is not a main segment. 
50 The same vector which was obtained as the 

motion vector of the "own" block by a prior art 
method is selected as the candidate for the main 
segment. 

Other segments in the block area which are not 
55 next to any segment of a neighboring block, which 
thus are not next to the border of the "own" block 
but entirely within the own block, get as the motion 
vector that motion vector, which in a known way 
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was obtained for the own block. 

Other segments in the block area which are 
next to a segment of a neighboring block, being 
however not the main segment of the neighboring 
block, get as the motion vector that motion vector, 
which in a known way was obtained for the own 
block. 

Other segments within the block area which are 
next to a main segment of a neighboring block get 
as the motion vector that motion vector value of the 
main segment in the neighboring block. This value 
thus equals the value of that value which was 
calculated by the prior art method for the motion 
vector of the neighboring block. 

Except for the main segment a segment can 
thus get as the motion vector either the motion 
vector of the own block or the motion vector of the 
neighboring block. 

Now each main segment has a candidate for 
the motion vector. The final motion vector Is de- 
cided so that we examine whether the main seg- 
ment continues into at least two neighboring 
blocks, and whether the part in the neighboring 
area is a main segment. In that case the segments 
form the same object, so that their motion vectors 
should be consistent. Therefore, a motion vector 
candidate is not accepted if it differs from the 
motion vectors of the corresponding segments In 
the neighboring blocks it, but the motion vector is 
selected to be the value of the motion vectors of 
the main segments of the neighboring blocks. 

According to a preferred embodiment the 
preprocessing is a histogram equalization, or a 
smoothening of the distribution. This means that 
within the window In use we order the samples 
according to the magnitude of the gray scale value, 
e.g. 0 to 256. The result could be thought of as a 
histogram having a very uneven row of columns. 
Then the row of columns Is leveled so that it is as 
uniform as possible, which results in an increased 
contrast and clearly pronounced contours. 

According to a preferred embodiment median 
filtering Is used as postprocessing. 

The Invention is illustrated by the enclosed 
example figures, in which 

figure 1 shows four blocks, for which the 
motion vectors are obtained by any 
known means; 
figure 2 shows the original picture sequence; 
figure 3 is an enlarged area of the sequence 
of figure 1 ; 

figure 4 shows the figure 3 after histogram 

equalization; 

figure 5 shows the figure 4 after quantiza- 
tion; 

figure 6 shows the figure 5 after postproces- 
sing; and 

figure 7 shows the steps of the method as a 



block diagram. 
The picture sequence of figure 2 shows a train 
passing a station. An area of the size 48*48 pixels 
is outlined with white borders. This covers four 

5 blocks of 16*16 pixels, each having a motion vector 
of its own which is determined by any block match- 
ing method. According to the invention the motion 
vectors are found for substantially smaller areas 
than for the said areas of the size 16*16. The area 

10 outlined with white borders in figure 2 is shown 
enlarged and in black and white in figure 3. With 
the aid of the following figures we examine how the 
different steps of the method affect this picture. 
According to the first step (step 21 in figure 7) 

75 of the method the whole figure 2 Is preprocessed. 
There a histogram equalization is first made on the 
whole figure 2. The samples of the picture are 
ordered according to the magnitude of the gray 
scale value, e.g. with samples of 8 bits the order is 

20 0 to 256. The result can be seen as a histogram, 
where the height of each column is proportional to 
the number of samples with that value. Thus the 
row of columns Is very uneven. The row of col- 
umns Is then leveled so that it will be as even as 

25 possible, whereby the gray scale value of certain 
samples is increased and that of others decreased. 
As a result there is an increased contrast in the 
picture and the contours of surfaces are clearly 
visible. The result is shown in figure 4. This figure 

30 shows how the area of figure 3 has changed after 
the histogram equalization. The contrast has In- 
creased and the surface contours of different areas 
are more clearly visible. 

After equalization there is performed a uniform 

35 quantization (step 22 in figure 7) of the whole 
picture. In which the picture of 256 gray levels (8 
bits) is quantified into a picture of 4 levels (2 bits). 
In figure 5 we can see the result for the four blocks 
of interest. Already the reduction of gray levels 

40 makes the segments clearly visible. However, we 
can clearly see small areas formed by a few pixels, 
where the gray scale slightly differs from their 
surrounding. In order to further simplify the picture 
and to equalize the segments we further perform a 

45 postprocessing (step 23 in figure 7) in order to 
replace the value of these pixels with the value of 
the surrounding pixels. Any median operation Is 
suitable for the postprocessing. In this example we 
used a 9 point median filter. The pixels in a 3*3 

50 window are supplied to a filter, providing an output 
value (= the median of the pixels) which replaces 
the value of the center pixel. The obtained result is 
shown in figure 6. When we compare this with the 
original unprocessed picture 3 we can clearly see 

55 that the picture has changed Into a set of adjacent 
segments with clear borders. Each segment forms 
thus a set of adjacent pixels with the same gray 
scale value. 
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For each of these segments we now determine 
its own motion vector (step 25 in figure 7). The grid 
drawn in figure 6 represents the block boundaries, 
with which the picture originally was divided, and 
thus each block then has a motion vector deter- 
mined by some block matching method. Let us 
now examine the segmented picture through the 
blocks. A segment is cut at the block border when 
it extends over a block border into another block. 
For example, at the top left there is a large light 
segment within the blocks A and B. The border line 
L1 between the blocks divides this segment into 
two independent segments 1 in block A and 2 in 
block B. 

Within the block area there can be segments of 
four kinds: 

1) One main segment, which is the segment 
within the block area having the largest number 
of pixels. The segment 1 in block A and the 
segment 2 in block B are of this kind. 

2) One or more segments, which are not the 
main segment and do not border on a segment 
of the adjacent block. The segment 3 in block is 
on this kind. 

3) One or more segments, which are not the 
main segment and border on the main segment 
of an adjacent block. The segments 5 and 6 in 
block A and the segments 4, 7 and 8 in block B 

are of this kind. 

4) One or more segments, which are not the 
main segment and border on a segment of the 
adjacent block, which is not the main segment. 
The segment 4 of block B is of this kind. 

Each vector is provided with a motion vector 
according the following rule: 

The same vector which was obtained as the motion 
vector of the "own" block by a prior art method is 
selected as the candidate for the main segment. 
The candidate vectors are shown by arrows in the 
main segments 1 and 2. Now the main segment of 
each block has a motion vector candidate. The 
correctness of the motion vector candidates is 
checked when they are selected for the blocks 
(step 24 in figure 7). The final motion vector is 
selected so that we examine whether the main 
segment extends into at least two neighboring seg- 
ments, and whether the part in the neighboring 
area is a main segment. If this is true, then the 
segments form the same object, and so their mo- 
tion vectors should be consistent. Therefore, if a 
motion vector candidate is different from the mo- 
tion vectors of said segments, it is not accepted, 
but the motion vector gets the motion vector value 
of the main segments of the neighboring blocks. 
For example, the main segment 1 of block A ex- 
tends both into block B and (we assume) into block 
D, and the part of the segment in these blocks is 
also the main segment of the block. If the direction 



of the vector candidate for the main segment in 
block B would be e.g. "up", the final vector is 
corrected to have the direction "left", so that the 
vectors of the main segments are consistent and all 

5 pixels of the object formed by them have the same 
direction of motion. 

Other segments in the block area which are not 
next to any segment of a neighboring block, which 
thus are not next to the border of the "own" block 

10 but entirely within the own block, get as the motion 
vector that motion vector, which in a known way 
was obtained for the own block. Thus the segment 
3 gets as its motion vector the same motion vector 
as segment 1 . 

75 Other segments in the block area which are 

next to a segment of a neighboring block, being 
however not the main segment of the neighboring 
block, get as the motion vector that motion vector, 
which in a known way was obtained for the own 

20 block. Thus the segment 4 in block B gets as its 
motion vector the motion vector of block B (and at 
the same time of the main segment 2 of this 
block). 

Other segments within the block area which are 

25 next to a main segment of a neighboring block get 
as the motion vector the motion vector value of the 
main segment in the neighboring block. Thus the 
segments 7 and 8 of block B get as their motion 
vector the value which was calculated for the 

30 neighboring block 0 (e.g. up to the right). 

According to prior art the motion vector of the 
segments 7 and 8 in block B would point to the 
left, but using the invention the motion vector 
points now "up to the right". Because the seg- 

35 ments 7 and 8 clearly belong to the main segment 
of block C and thus also to its motion, the use of a 
known technique in picture processing would pro- 
duce obvious errors at the edges of an object. No 
errors are now generated because these are cor- 

40 rectly processed according to the invention. 

Each segment of the picture gets a motion 
vector according to the presented principles. 

The method is particularly well suited to be 
used in television systems where the picture is 

45 differently encoded at the originating end of the 
transmission link depending on its motion contents. 
In addition to the encoded picture also motion 
vectors of the segments are transmitted, so that the 
receiver is able to process the video signal dif- 

50 ferently depending on its motion contents. 

The method of the invention is based on the 
perception that the picture is processed in order to 
obtain distinctly discernible segments, whereby an 
object and the background are separated and form 

55 a surface of uniform shade without any details. 
Then they can be separately processed and be 
given motion vectors. Thus the invention does not 
in any way limit the methods by which each in- 
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dividual step of the method is realized. The above 
mentioned histogram equalization and median fil- 
tering are only some of the preferred embodi- 
ments, but they do not limit the invention in any 
way. 5 

Claims 

1. A method to determine the motion vectors of a 

television picture when the picture first has io 
been divided into blocks of equal size and 
form, and when for each block a motion vector 
has been calculated which represents the mag- 
nitude and direction of motion contents for all 
pixels of the block, characterized in that 75 

- the television picture is preprocessed in 
a process changing the gray scale values 
of the pixels in order to equalize their 
value distribution; 

- the preprocessed picture is quantified in 20 
order to reduce the number of its gray 
levels, whereby a picture is obtained 
comprising uniformly shaded segments 

of different sizes and different forms; 

- the block motion vector value of respec- 25 
five block is made the motion vector 
candidate for the first main segment of 

the largest segment in each block area; 

- if the main segment extends over at least 

two neighboring blocks and the part in 30 
those block areas is also the main seg- 
ment of the respective neighboring block, 
then the motion vector candidate is re- 
jected and the main segment gets as its 
motion vector the block motion vector of 35 
the neighboring block, otherwise the mo- 
tion vector candidate is accepted as the 
motion vector of the main segment; 

- other such segments within the block, 
which border on the main segment of the 40 
neighboring block, get as the motion vec- 
tor the motion vector value of this neigh- 
boring block. 



7. 



The method according to claim 1 or 2, char- 
acterized in that the preprocessing is a histo- 
gram equalization. 

The method according to claim 1 or 2, char- 
acterized in that the quantization is equidistant 
and that 8 bit pixels are quantified into 2 bit 
pixels. 

The method according to claim 1 or 2, char- 
acterized in that further the quantified picture 
is filtered. 

The method according to claim 5, character- 
ized in that the filtering is a median filtering. 

The method according to claim 6, character- 
ized In that It uses a 9 point median filter. 



2. The method according to claim 1, character- 45 
ized in that 

- other segments in the block area which 

are not next to any segment of a neigh- 
boring block get as the motion vector the 
motion vector value of the respective so 
block; and 

- other segments in the block area, which 
are next to a segment of a neighboring 
block which is not the main segment of 

the neighboring block, get as the motion 55 
vector the motion vector value of the 
respective block. 
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